Image processing apparatus, image generating method, and storage medium

ABSTRACT

It is an object of the present invention to reduce a possibility that a subject (specific object) of a virtual viewpoint image and a virtual object area overlap. To achieve the object, the present invention comprises a first identifying unit for identifying a three-dimensional position of the specific object captured respectively from different directions by a plurality of cameras; a second identifying unit for identifying a three-dimensional position of a virtual viewpoint related to generation of the virtual viewpoint image based on captured images obtained by the plurality of cameras; and a determining unit for determining a position of the virtual object area to which a virtual object to be displayed in the virtual viewpoint image is disposed, based on the three-dimensional position of the specific object identified by the first identifying unit and the three-dimensional position of the virtual viewpoint identified by the second identifying unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Patent ApplicationNo. PCT/JP2017/030092, filed Aug. 23, 2017, which claims the benefit ofJapanese Patent Application No. 2016-179529, filed Sep. 14, 2016, andJapanese Patent Application No. 2017-124110, filed Jun. 26, 2017, all ofwhich are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, an imagegenerating method, and a storage medium.

Description of the Related Art

In recent years, a technique of installing a plurality of cameras atdifferent positions, performing synchronous image capture with multipleviewpoints, and generating not only images at the camera installationpositions but also virtual viewpoint images from arbitrary viewpoints byusing a plurality of viewpoint images obtained by the capture attractsattention. Here, it is possible to realize the generation of the virtualviewpoint images based on the plurality of viewpoint images and browsingof the generated images by collecting the images captured by theplurality of cameras in an image processing unit such as a server or thelike, performing a process such as rendering or the like to thecollected images based on virtual viewpoints in the image processingunit, and displaying the obtained virtual viewpoint images on a user'sviewing terminal.

Japanese Patent Application Laid-Open No. 2014-41259 discloses atechnique of providing an advertisement frame in a predetermined areasuch as a wall surface of a stadium in soccer-stadium relay broadcast orthe like, and displaying advertisement information corresponding to aviewing environment to the advertisement frame in video data of avirtual viewpoint including the advertisement frame.

However, there is a fear that an advertisement area overlaps a subjectsuch as a soccer player or the like. For example, if the subject and theadvertisement area overlap each other due to a movement of the subject,there is a fear that an advertisement is not displayed. Besides, forexample, if the subject and the advertisement area overlap each otherdue to a movement of a virtual viewpoint, there is a fear that theadvertisement is not displayed. A similar problem may arise in a casewhere not only the advertisement but also another virtual object isdisplayed in a virtual viewpoint image.

The present invention has been completed in view of such problems, andit is an object thereof to reduce a possibility that the subject(specific object) of the virtual viewpoint image and a virtual objectarea overlap each other.

SUMMARY OF THE INVENTION

Therefore, the present invention is characterized by comprising: a firstidentifying unit configured to identify a three-dimensional position ofa specific object captured respectively from different directions by aplurality of cameras; a second identifying unit configured to identify athree-dimensional position of a virtual viewpoint related to generationof a virtual viewpoint image based on captured images obtained by theplurality of cameras; and a determining unit configured to determine aposition of a virtual object area to which a virtual object to bedisplayed in the virtual viewpoint image is disposed, based on thethree-dimensional position of the specific object identified by thefirst identifying unit and the three-dimensional position of the virtualviewpoint identified by the second identifying unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for describing an image processing system.

FIG. 2 is a diagram for describing an installation example of imagecapturing apparatuses.

FIG. 3 is a diagram for describing a hardware constitution of an imagegenerating apparatus.

FIG. 4 is a diagram for describing a functional configuration of theimage generating apparatus.

FIG. 5 is a flowchart for describing an image process.

FIG. 6A is an explanatory diagram of an advertisement area determiningprocess.

FIG. 6B is an explanatory diagram of the advertisement area determiningprocess.

FIG. 7A is an explanatory diagram of the advertisement area determiningprocess.

FIG. 7B is an explanatory diagram of the advertisement area determiningprocess.

FIG. 8A is an explanatory diagram of a modified example.

FIG. 8B is an explanatory diagram of the modified example.

FIG. 9 is a diagram for describing a functional configuration of theimage generating apparatus according to a second embodiment.

FIG. 10 is a flowchart for describing an image process according to thesecond embodiment.

FIG. 11A is an explanatory diagram of an advertisement area determiningprocess.

FIG. 11B is an explanatory diagram of the advertisement area determiningprocess.

FIG. 12 is a diagram for describing a functional configuration of theimage generating apparatus according to a third embodiment.

FIG. 13 is a flowchart for describing an image process according to thethird embodiment.

FIG. 14 is a diagram for describing a functional configuration of theimage generating apparatus according to a fourth embodiment.

FIG. 15 is a flowchart for describing an image process according to thefourth embodiment.

FIG. 16 is an explanatory diagram of an advertisement area identifyingprocess according to the fourth embodiment.

FIG. 17 is an explanatory diagram of the advertisement area identifyingprocessing according to the fourth embodiment.

FIG. 18A is an explanatory diagram of a generating process of anadvertisement-synthesized virtual viewpoint image according to thefourth embodiment.

FIG. 18B is an explanatory diagram of the generating process of theadvertisement-synthesized virtual viewpoint images according to thefourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed with reference to the attached drawings.

First Embodiment

FIG. 1 is a diagram for describing an image processing system 100according to the present embodiment. The image processing system 100comprises a plurality of image capturing apparatuses 110, an imagegenerating apparatus 120 which is an example of an image processingapparatus, and a terminal apparatus 130. Each of the image capturingapparatuses 110 and the image generating apparatus 120 are connected viaa communication cable such as a LAN (Local Area Network) cable or thelike. In the present embodiment, it is assumed that the communicationcable is the LAN cable, but the communication cable is not limited tothat in the embodiment. As another example, it is assumed that thecommunication cable may be a video transmission cable such as aDisplayPort, an HDMI™ (High Definition Multimedia Interface) or thelike.

The image capturing apparatus 110 is a digital camera capable ofcapturing an image (still image and moving image), for example. Eachimage capturing apparatus 110 is installed so as to surround a stadiumsuch as a soccer field or a specific subject, and captures an image(video). The captured image is transmitted from the image capturingapparatus 110 to the image generating apparatus 120. FIG. 2 is a diagramfor describing an installation example of the image capturingapparatuses 110. In the present embodiment, it is assumed that theplurality of image capturing apparatuses 110 are installed so as tocapture the whole or a part of the soccer field. That is, the imageprocessing system 100 according to the present embodiment includes theplurality of image capturing apparatuses 110 for capturing a subjectrespectively from a plurality of directions.

The image generating apparatus 120 accumulates the captured imagesobtained by the image capturing apparatuses 110, and, when virtualviewpoint information is input on the terminal apparatus 130 by a user'soperation, generates a virtual viewpoint image based on the capturedimage and a virtual viewpoint. Here, the virtual viewpoint informationis information indicating a three-dimensional position of a virtuallyset viewpoint (virtual viewpoint) in a virtual space constructed fromthe captured images. It is assumed that the virtual viewpointinformation includes at least position information of a relativeposition with respect to a predetermined position such as the center ofthe captured stadium or the like, that is, position information of frontand rear, left and right, and right and left with respect to thepredetermined position, and direction information of a direction fromthe predetermined position, that is, direction information of anglesaround front and rear, left and right, and up and down.

The image generating apparatus 120 is, for example, a server apparatus,and has a database function and an image processing function. In adatabase, as a background image, an image obtained by capturing a sceneof a competition venue where a subject does not exist beforehand becausea game is not yet started is previously held via the image capturingapparatus 110. Besides, in a scene in which a subject such as acompetitor during a competition exists, a foreground of a specificobject or the like to be the subject is separated by an image processand held as a specific object image. Incidentally, the specific objectmay be not only the competitor during the competition but also anotherspecific person (for example, a reserve, a director, a referee, etc.).Also, the specific object may be an object such as a ball or a goal forwhich an image pattern is predetermined.

It is assumed that the virtual viewpoint image corresponding to thevirtual viewpoint information is generated from the background image andthe specific object image managed in the database. For example,model-based rendering (MBR) is used as a generating method of thevirtual viewpoint image. The MBR is a method of generating the virtualviewpoint image by using a three-dimensional model generated based on aplurality of captured images obtained by capturing the subject from aplurality of directions. More specifically, the MBR is a technique ofgenerating, as an image, appearance of a scene from the virtualviewpoint by using a three-dimensional shape (model) of a target sceneobtained by a three-dimensional shape restoring method such as a visualhull, a multi-view-stereo (MVS) or the like. Incidentally, as thegenerating method of the virtual viewpoint image, a rendering methodother than the MBR may be used. The generated virtual viewpoint image istransmitted to the terminal apparatus 130 via a LAN cable or the like.

The terminal apparatus 130 is, for example, a PC (Personal Computer) ora tablet. A controller 131 is, for example, a mouse, a keyboard, asix-axis controller and a touch panel, and a user performs operations byusing them, and displays a still image or a moving image on a screen.For example, the terminal apparatus 130 displays the virtual viewpointimage received from the image generating apparatus 120 on a displayscreen 132. The terminal apparatus 130 further accepts an instruction tomove the virtual viewpoint (instruction related to movement amount andmovement direction) in accordance with the user's operation with respectto the connected controller 131, and transmits a transmission signalindicating instruction information according to the accepted instructionto the image generating apparatus 120.

Incidentally, in the present embodiment, an example in which the virtualviewpoint image generated based on the virtual viewpoint set by theterminal apparatus 130 is displayed on the terminal apparatus 130 willbe mainly described, but the present invention is not limited thereto.For example, the virtual viewpoint image generated based on the virtualviewpoint set by the terminal apparatus 130 may be displayed by anotherterminal apparatus 130 (e.g., smartphone) or the like.

FIG. 3 is a diagram for describing a hardware constitution of the imagegenerating apparatus 120. The image generating apparatus 120 comprises aCPU 301, a ROM 302, a RAM 303, an HDD 304, a displaying unit 305, aninputting unit 306, and a communicating unit 307. The CPU 301 readscontrol programs stored in the ROM 302, and executes various processes.The RAM 303 is used as a temporary storage area such as a main memory, awork area and the like of the CPU 301. The HDD 304 stores various data,various programs and the like. The displaying unit 305 displays variouskinds of information. The inputting unit 306 includes a keyboard and amouse, and accepts various operations by the user. The communicatingunit 307 performs a communication process with an external apparatussuch as an image forming apparatus via a network. Incidentally, as thenetwork, the Ethernet™ can be used. As another example, thecommunicating unit 307 may communicate with an external apparatuswirelessly.

The functions and processes of the image generating apparatus 120, whichwill be described later using flowcharts and the like, are realized bythe CPU 301 which reads programs stored in the ROM 302 or the HDD 304and executes the read programs. However, it is also possible to realizeall or part of the functions and processes of the image generatingapparatus 120 by dedicated hardware. In this case, the CPU 301 of theimage generating apparatus 120 controls the dedicated hardware, therebyrealizing each function and process described later. Besides, thehardware constitution of the terminal apparatus 130 is similar to thehardware constitution of the image generating apparatus 120. That is,the functions and processes of the terminal apparatus 130, which will bedescribed later using the flowcharts and the like, may be realized bythe CPU 301, or part or whole of the functions and processes thereof maybe realized by dedicated hardware.

FIG. 4 is a diagram for describing a functional configuration of theimage generating apparatus 120. The image generating apparatus 120comprises a communication processing unit 401, a separating unit 402, aseparation image storing unit 403, a virtual viewpoint informationgenerating unit 404, and an image obtaining unit 405. The imagegenerating apparatus 120 further comprises an object identifying unit406, an advertisement area determining unit 407, an advertisement imagestoring unit 408, a virtual advertisement image generating unit 409, anda virtual viewpoint image generating unit 410.

The communication processing unit 401 controls transmission andreception of information between the image generating apparatus 120 andthe terminal apparatus 130. For example, the communication processingunit 401 converts a transmission signal received from the terminalapparatus 130 into instruction information. For example, the instructioninformation is user operation information which is composed of changeamounts of position information (x, y, z) indicating a position of thevirtual viewpoint in the virtual viewpoint image and directioninformation (rx, ry, rz) indicating a virtual capturing direction. Thecommunication processing unit 401 also converts a transmission signalreceived from the image capturing apparatus 110 into data of a capturedimage.

The separating unit 402 separates, from the captured images received viathe communication processing unit 401, an image obtained by capturing ascene of a competition venue in a state that a specific subject(specific object) does not exist because a game is not yet started, andrecords the separated image as the background image in the separationimage storing unit 403. Further, the separating unit 402 extracts thespecific subject such as a player or the like from an image capturedduring a competition, and records an image of the extracted specificsubject as a specific object image. The separation image storing unit403 stores the background image and the specific object image.Incidentally, the separation image storing unit 403 stores thebackground image and the specific object image together with capturingposition information indicating a capturing position (three-dimensionalposition) in the world coordinate system (three-dimensional coordinatesystem) of a capturing space. That is, the three-dimensional position isdefined by a position in each direction corresponding to a height, awidth and a depth in the capturing space. The capturing positioninformation with respect to the world coordinate system is obtained, forexample, by capturing a marker or the like previously set and performingthree-dimensional image measurement or the like of the marker.

The virtual viewpoint information generating unit 404 generates virtualviewpoint information (x, y, z, rx, ry, rz) from change amounts of theposition and the direction included in the instruction informationaccepted by the communication processing unit 401. Here, the virtualviewpoint information is information obtained by adding or subtractingthe change amount included in the instruction information to or from thevirtual viewpoint information before change, using, e.g., the center ofthe stadium as the origin. The image obtaining unit 405 obtains from theseparation image storing unit 403 a plurality of specific object imagesand background images corresponding to the virtual viewpoint informationgenerated by the virtual viewpoint information generating unit 404.Based on the specific object image obtained by the image obtaining unit405, the object identifying unit 406 identifies a position and a size ofthe specific object in the capturing space. The object identifying unit406 generates specific object information indicating the position andthe size of the specific object.

Based on the virtual viewpoint information generated by the virtualviewpoint information generating unit 404 and the specific objectinformation generated by the object identifying unit 406, theadvertisement area determining unit 407 determines a position(arrangement position) at which an advertisement area is disposed in thecapturing space. Here, the advertisement area is an area for displayingan advertisement image. The advertisement image storing unit 408 storesthe advertisement image. Here, the advertisement image is an imageincluding a company name logo, a product logo, a product image and thelike. Incidentally, the advertisement image is not limited to that inthe embodiment. The virtual advertisement image generating unit 409disposes the advertisement image stored in the advertisement imagestoring unit 408, in the advertisement area determined by theadvertisement area determining unit 407, thereby generating a virtualadvertisement image in which an advertisement is disposed at apredetermined virtual position as seen from the virtual viewpoint.

Incidentally, in the present embodiment, the virtual viewpoint image inwhich the advertisement image is disposed is generated, but the image tobe disposed in the virtual viewpoint image is not limited to theadvertisement image, and another virtual object may be used. Here, theadvertisement image is an example of a virtual object, and theadvertisement area is an example of a virtual object area.

The virtual viewpoint image generating unit 410 generates the virtualviewpoint image based on the specific object image and the backgroundimage obtained by the image obtaining unit 405 and the virtualadvertisement image generated by the virtual advertisement imagegenerating unit 409. That is, the virtual viewpoint image generated bythe virtual viewpoint image generating unit 410 includes theadvertisement image. The virtual viewpoint image generated by thevirtual viewpoint image generating unit 410 is converted into atransmission signal which can be transmitted by the communicationprocessing unit 401, and transmitted to the terminal apparatus 130.

FIG. 5 is a flowchart for describing an operation in image process to beperformed by the image generating apparatus 120. In S501, thecommunication processing unit 401 confirms whether or not theinstruction information is received from the terminal apparatus 130.When receiving the instruction information (YES in S501), thecommunication processing unit 401 advances the process to S502. In S502,the virtual viewpoint information generating unit 404 generates thevirtual viewpoint information based on the instruction information.Next, in S503, the image obtaining unit 405 obtains the plurality ofspecific object images and the background image necessary for generatingthe virtual viewpoint image corresponding to the virtual viewpointinformation from the separation image storing unit 403. Incidentally,the image obtaining unit 405 obtains image data obtained from at leastthe two or more image capturing apparatuses 110 which are close to theposition of the virtual viewpoint information.

Next, in S504, the object identifying unit 406 identifies the positionand the size of the specific object in the capturing space by using athree-dimensional image analyzing method with respect to the imagescaptured by the plurality of image capturing apparatuses 110.Incidentally, the process for identifying the position and the size ofthe specific object is not limited to that in the embodiment.

Next, in S505, the advertisement area determining unit 407 determinesthe arrangement position of the advertisement area based on the specificobject information and the virtual viewpoint information. Morespecifically, the advertisement area determining unit 407 determines aposition at which the specific object image and the virtualadvertisement image do not overlap in the virtual viewpoint image, asthe arrangement position of the advertisement area. This process will bedescribed later with reference to FIGS. 6A and 6B and FIGS. 7A and 7B.Next, in S506, the virtual advertisement image generating unit 409disposes the advertisement image stored in the advertisement imagestoring unit 408, at the arrangement position of the advertisement areadetermined by the advertisement area determining unit 407, therebygenerating the virtual advertisement image.

Next, in S507, the virtual viewpoint image generating unit 410 generatesthe virtual viewpoint image based on the specific object image and thebackground image obtained by the image obtaining unit 405. Next, inS508, the virtual viewpoint image generating unit 410 generates thevirtual viewpoint image in which the advertisement is disposed, bysynthesizing the virtual viewpoint image generated in S507 and thevirtual advertisement image generated in S506. Next, in S509, thecommunication processing unit 401 transmits (outputs) the virtualviewpoint image (display image) obtained in S508 to the terminalapparatus 130. Thus, the image process ends.

FIGS. 6A and 6B and FIGS. 7A and 7B are the explanatory diagrams of theadvertisement area determining process (S505). In FIG. 6A, a virtualviewpoint 600, a virtual capturing range 601, a specific object 602 andan advertisement area 603 are conceptually shown. In the example of FIG.6A, as seen from the virtual viewpoint 600, the position on the rightside of the virtual capturing range 601 and behind in the capturingdirection relative to the specific object 602 is determined as thearrangement position of the specific object 602. FIG. 6B is the diagramfor describing a virtual viewpoint image 610 corresponding to FIG. 6A.In the virtual viewpoint image 610, a virtual advertisement image 612 isdisposed on the left side of the specific object image 611 withoutoverlapping a specific object image 611.

Further, as shown in FIG. 7A, it is assumed that a specific object 702is positioned on the left side of a capturing range 701. In this case,the advertisement area determining unit 407 determines the position onthe right side of and at the rear of the specific object 702 as theposition of an advertisement area 703. FIG. 7B is the diagram fordescribing a virtual viewpoint image 710 corresponding to FIG. 7A. Inthe virtual viewpoint image 710, a virtual advertisement image 712 isdisposed on the right side of a specific object image 711 withoutoverlapping the specific object image 711.

As described above, the image generating apparatus 120 according to thefirst embodiment determines the position not overlapping the specificobject (subject) as the advertisement area, and disposes the virtualadvertisement image in the advertisement area. More specifically, theimage generating apparatus 120 determines the advertisement area basedon the position of the virtual viewpoint in the three-dimensional spaceand the position of the specific object in the three-dimensional space.Incidentally, the three-dimensional space in the present embodiment is aspace which is identified by, for example, a height direction (gravitydirection), a width direction and a depth direction in a space (forexample, a stadium such as a soccer field) being a capturing target bythe plurality of image capturing apparatuses 110. This three-dimensionalspace can be represented by the three-dimensional coordinates using, forexample, the center of the stadium as the origin. According to the aboveconstitution, it is possible to prevent that a part or whole of theadvertisement image is not displayed because the subject overlaps theadvertisement image. That is, it is possible to reduce a loss of displayopportunity of the advertisement image. In other words, the imagegenerating apparatus 120 can appropriately dispose the advertisementimage in the virtual viewpoint image.

As a first modified example, the image generating apparatus 120 maychange the position of the advertisement area according to a movement ofthe specific object. More specifically, the image generating apparatus120 detects the movement of the object based on each frame of video. Asshown in FIG. 8A, when a movement of a specific object 802 in thedirection approaching a virtual viewpoint 800 is detected, the imagegenerating apparatus 120 changes the position of an advertisement area803 to a further rear position.

FIG. 8B is a diagram for describing a change of the virtual viewpointimage according to the movement of the specific object 802. Inaccordance with the movement of the specific object 802, the virtualviewpoint image changes from the virtual viewpoint image 810 to thevirtual viewpoint image 820. That is, a virtual viewpoint image 810corresponds to the state before the movement of the specific object 802in FIG. 8A, and a virtual viewpoint image 820 corresponds to the stateafter the movement of the specific object 802 in FIG. 8A. A specificobject image 821 of the virtual viewpoint image 820 is displayed largerthan a specific object image 811 of the virtual viewpoint image 810. Onthe other hand, a virtual advertisement image 822 of the virtualviewpoint image 820 is displayed smaller than a virtual advertisementimage 812 of the virtual viewpoint image 810. Moreover, in this case,the image generating apparatus 120 may perform an image process ofgiving a sense of distance, such as blurring, shading or the like to thevirtual advertisement image 822 as compared with the virtualadvertisement image 812.

More specifically, the image generating apparatus 120 determines theposition of the advertisement area in the capturing direction accordingto the distance between the virtual viewpoint and the specific object.For example, as the distance between the virtual viewpoint and thespecific object becomes shorter, the image generating apparatus 120determines a further rear position as the position of the advertisementarea. Incidentally, it is assumed that the image generating apparatus120 determines the position of the advertisement area from, for example,a relational expression indicating a relationship between the distancebetween the virtual viewpoint and the specific object and the positionof the advertisement area. In this case, it is assumed that relationalexpression is previously stored in the HDD 304 of the image generatingapparatus 120. As another example, the image generating apparatus 120may determine the position of the advertisement area with reference to acorrespondence table in which the distance between the virtual viewpointand the specific object and the position of the advertisement area areassociated with each other. In this case, it is assumed that thecorrespondence table is previously stored in the HDD 304 of the imagegenerating apparatus 120. Besides, at this time, the image generatingapparatus 120 may also change a position in another direction inaddition to the position in the capturing direction of the advertisementarea, in accordance with the distance between the virtual viewpoint andthe specific object.

In this way, as the specific object approaches the virtual viewpoint,the image generating apparatus 120 changes the position of theadvertisement area to a further rear position. Thus, it is possible toprevent that a part or whole of the virtual advertisement image is notdisplayed even in the case where the specific object is moving in adirection approaching the virtual viewpoint.

The image generating apparatus 120 detects the movement direction of thespecific object, and changes the position of the advertisement areaaccording to the detected result. However, this operation is not limitedto the case where the specific object moves toward the virtualviewpoint. For example, in a case where the specific object moves in thedirection of the advertisement area, the image generating apparatus maychange the position of the advertisement to a further rear position, ormay change the position of the advertisement area in a direction awayfrom the specific object.

As a second modified example, instead of determining the arrangementposition of the advertisement area in the capturing space, the imagegenerating apparatus 120 may determine an arrangement position of theadvertisement area in the virtual viewpoint image. In this case, theimage generating apparatus 120 may generates a virtual viewpoint imagenot including an advertisement image based on the virtual viewpointinformation, and determine a position which does not overlap thespecific object image in the virtual viewpoint image as the position ofthe advertisement area. Then, the image generating apparatus 120 obtainsa virtual viewpoint image on which the advertisement image is displayed,by superimposing the virtual advertisement image on the advertisementarea. Further, the image generating apparatus 120 may determine the sizeof the advertisement area in addition to the arrangement position of theadvertisement area in the virtual viewpoint. For example, in a casewhere a space for disposing the advertisement area of an initial sizecannot be secured, the image generating apparatus 120 may set the sizeof the advertisement area to a smaller size. According to the secondmodified example, rather than determining the arrangement position ofthe advertisement area in a three-dimensional space, it is possible toreduce a processing load related to determination of the advertisementarea based on the position of the virtual viewpoint in thethree-dimensional space and the position of the specific object.

As a third modified example, instead of generating the virtual viewpointinformation according to the user's operation, the image generatingapparatus 120 may automatically generate such virtual viewpointinformation as tracking a specific object such as a specific player, aball or the like.

As a fourth modified example, the image processing apparatus 120 maysequentially change the position of the advertisement area for eachframe of a moving image as the virtual viewpoint image, or afterchanging the position once, may fix the position of the advertisementarea for a predetermined period.As a fifth modified example, the image generating apparatus 120 maydirectly generate a virtual viewpoint image into which an advertisementis inserted, based on the images of the specific object, the backgroundand the advertisement, and the respective dispositions thereof in thethree-dimensional space. According to such a method, as compared with acase where the virtual viewpoint image in which an advertisement is notinserted and the virtual advertisement image are once generated and thensynthesized, it is possible to reduce the whole processing amount of theimage generating apparatus 120.As a sixth modified example, the image generating apparatus 120 mayoutput the virtual viewpoint image in which an advertisement is notinserted and information indicating the position of the advertisementarea determined by the advertisement area determining unit 407 to anexternal apparatus. Then, the external apparatus may insert theadvertisement into the position of the advertisement area indicated bythe information obtained from the image generating apparatus 120, andgenerate the virtual viewpoint image in which the advertisement has beeninserted.

Second Embodiment

Next, the image processing system according to the second embodimentwill be described. FIG. 9 is a diagram for describing a functionalconfiguration of an image generating apparatus 900 according to thesecond embodiment. The image generating apparatus 900 further comprisesa background shape identifying unit 901 in addition to the configurationof the image generating apparatus 120 of the first embodiment describedwith reference to FIG. 4.

The background shape identifying unit 901 identifies a shape of thebackground based on the plurality of background images obtained by theimage obtaining unit 405. Then, the background shape identifying unit901 generates background shape information which indicates the shape ofthe background such as the disposition of the background, the positioninformation of the feature point, and the like. Incidentally, as amethod of identifying the shape of the background, there is a process ofidentifying the shape including the disposition and the depth of thebackground by using the three-dimensional image analyzing method withrespect to images captured by a plurality of cameras. However, anothermethod may be used. An advertisement area determining unit 902determines the arrangement position of the advertisement area in thecapturing space based on the specific object information generated bythe object identifying unit 406, the background shape information, andthe virtual viewpoint information.

FIG. 10 is a flowchart for describing an image process to be performedby the image generating apparatus 900 according to the secondembodiment. It should be noted that, among the processes shown in FIG.10, the same processes as those of the image process shown in FIG. 5 aredenoted by the same reference numerals respectively, and the descriptionthereof will be omitted. After the process in S504, in S1001, thebackground shape identifying unit 901 identifies the shape of thebackground based on the plurality of background images obtained by theimage obtaining unit 405. Then, the background shape identifying unit901 generates the background shape information which indicates the shapeof the background such as the disposition of the background, theposition information of the feature point, and the like.

Next, in S1002, the advertisement area determining unit 902 determinesthe arrangement position of the advertisement area based on the specificobject information generated by the object identifying unit 406, thebackground shape information, and the virtual viewpoint information.More specifically, the advertisement area determining unit 902 firstdetermines a position which does not overlap the specific object as aprovisional arrangement position of the advertisement area withoutconsidering the shape of the background. Then, the advertisement areadetermining unit 902 determines a position of a predetermined shape onthe straight line passing through the virtual viewpoint and theprovisional arrangement position, as the arrangement position. Here, thepredetermined shape includes a surface facing the virtual viewpointside, or the like. The surface facing the virtual viewpoint sideincludes a wall surface. Besides, as another example, the advertisementarea determining unit 902 may project the advertisement area on thebackground shape such that the virtual viewpoint image seems to facedirectly when seen from the virtual viewpoint. After the process ofS1002, the CPU 301 advances the process to S506.

FIGS. 11A and 11B are explanatory diagrams of the advertisement areadetermining process (S1002). FIG. 11A conceptually shows the virtualviewpoint 600, the virtual capturing range 601, the specific object 602and the advertisement area 603. In the background, a wall surface 1101and a signboard 1102 exist as the background. In this case, theadvertisement area determining unit 902 determines the position of thesignboard 1102 existing at a position not overlapping the image of thespecific object 602 in the virtual viewpoint image, as the position ofthe advertisement area. FIG. 11B is a diagram for describing a virtualviewpoint image 1110 corresponding to FIG. 11A. In the virtual viewpointimage 1110, a virtual advertisement image 1112 is disposed on thesignboard 1102 in the background image on the left side of a specificobject image 1111, without overlapping the specific object image 1111.

Besides, in case of a virtual viewpoint of which the capturing directionis overlooking the stadium from just above the stadium, the imagegenerating apparatus 900 determines the ground as the advertisementarea. Thus, it is possible to generate the virtual viewpoint imagehaving no adverse effect on competition viewing and having anadvertisement effect. Incidentally, other configurations and processesof the image processing system according to the second embodiment aresimilar to those of the image processing system according to the firstembodiment.

As described above, in the second embodiment, the image generatingapparatus 900 determines the position of the advertisement area inconsideration of the shape of the background. Therefore, it is possibleto prevent that the virtual advertisement image is disposed behind thebackground such as the wall, the signboard or the like, depending on theposition and shape of the background. Besides, since the imagegeneration apparatus 900 determines the arrangement position of theadvertisement area based on the three-dimensional shape such as the wallsurface, the signboard and the like of the background, it is possible togenerate a virtual viewpoint image for which the user does not feeluncomfortable.

Third Embodiment

Next, the image processing system according to the third embodiment willbe described. FIG. 12 is a diagram for describing a functionalconfiguration of an image generating apparatus 1200 according to thethird embodiment. The image generating apparatus 1200 further comprisesan object analyzing unit 1201 in addition to the configuration of theimage generating apparatus 120 of the first embodiment described withreference to FIG. 4. Based on the virtual viewpoint informationgenerated by the virtual viewpoint information generating unit 404, theobject analyzing unit 1201 generates a virtual viewpoint image notincluding an advertisement image. Then, the object analyzing unit 1201calculates a percentage of the size of the specific object image to thewhole virtual viewpoint image, and generates percentage informationindicating the calculated percentage.

An advertisement area determining unit 1202 determines an arrangementposition of the advertisement area in the capturing space based on thevirtual viewpoint information, the specific object information, and thepercentage information. Incidentally, in the present embodiment, it isassumed that an initial position of the advertisement area is preset.With reference to the initial position, the advertisement areadetermining unit 1202 determines the position in the capturing directionbased on the percentage information.

For example, it is assumed that the whole virtual viewpoint image is thearea of (0, 0) to (1920, 1080) dots and the specific object image is thearea of (960, 0) to (1560, 1080) dots. In this case, when the percentageexceeds a preset threshold such as 30% or the like, the advertisementarea determining unit 1202 determines a rear position by a predetermineddistance from the initial position in the capturing direction, as thearrangement position of the advertisement area. Here, for example, thepredetermined distance is a distance which doubles the distance from thevirtual viewpoint. Incidentally, when determining the arrangementposition of the advertisement area, the advertisement area determiningunit 1202 may use a relational expression indicating the relationshipbetween the percentage and the arrangement position of the advertisementarea. As another example, the advertisement area determining unit mayuse a correspondence table on which the percentage and the arrangementposition of the advertisement area are associated with each other.

FIG. 13 is a flowchart for describing an image process to be performedby the image generating apparatus 1200 according to the thirdembodiment. It should be noted that, among the processes shown in FIG.13, the same processes as those of the image process shown in FIG. 5 aredenoted by the same reference numerals respectively, and the descriptionthereof will be omitted. After the process in S504, in S1301, the objectanalyzing unit 1201 generates a virtual viewpoint image not including anadvertisement image based on the virtual viewpoint information. Then,the object analyzing unit 1201 calculates a percentage of the size ofthe specific object image to the whole virtual viewpoint image, andgenerates percentage information indicating the calculated percentage.

Next, in S1302, the advertisement area determining unit 1202 determinesthe arrangement position of the advertisement area in the capturedimage, based on the virtual viewpoint information, the specific objectinformation, and the percentage information. Thereafter, the CPU 301advances the process to S506. Other configurations and processes of theimage processing system according to the third embodiment are similar tothose of the image processing system according to another embodiment.

As described above, in the third embodiment, the image generatingapparatus 1200 determines the arrangement position of the advertisementarea based on the percentage of the specific object image to the wholeimage. Therefore, for example, as in the virtual viewpoint image 820shown in FIG. 8B, even when the specific object moves toward the virtualviewpoint and the specific object image becomes large, it is possible todispose the virtual advertisement image to a position not overlappingthe specific object image. Besides, even for viewpoint zooming, it ispossible to dispose the virtual advertisement image at a position notoverlapping the specific object image in the virtual viewpoint image tobe operated by the user.

Fourth Embodiment

Next, the image processing system according to the fourth embodimentwill be described. FIG. 14 is a diagram for describing a functionalconfiguration of an image generating apparatus 1400 according to thefourth embodiment. The image generating apparatus 1400 comprises anadvertisement area identifying unit 1401, an advertisement imageconverting unit 1402 and an image synthesizing unit 1403, instead of theobject identifying unit 406, the advertisement area determining unit 407and the virtual advertisement image generating unit 409 of the imagegenerating apparatus 120 according to the first embodiment.

The advertisement area identifying unit 1401 identifies whether or notthe advertisement area is included in the virtual viewpoint imagegenerated by the virtual viewpoint image generating unit 410. Morespecifically, the advertisement area identifying unit 1401 identifiesthe advertisement area in the virtual viewpoint image by performingtwo-dimensional projection transformation to a predeterminedadvertisement area in a three-dimensional virtual viewpoint space basedon the virtual viewpoint information. Incidentally, it is assumed that aposition, a size and a shape of the advertisement area in the virtualviewpoint space are preset. In the present embodiment, it is assumedthat the advertisement area is a rectangle, and the width and the heightthereof are set as the size and the shape. When the advertisement areais included in the virtual viewpoint image, the advertisement areaidentifying unit 1401 generates advertisement area informationindicating the position, the width and the height of the advertisementarea with respect to the whole virtual viewpoint image. It should benoted that one or more advertisement areas may be set in the virtualviewpoint space. As another example, the advertisement area may bedynamically set according to the virtual viewpoint.

Besides, the process to be performed by the advertisement areaidentifying unit 1401 for identifying the advertisement area is notlimited to that in the embodiment.

As another example, a predetermined pattern object such as, e.g., QRCode™ (Quick Response code) is disposed in the advertisement area. Then,the advertisement area identifying unit 1401 may identify theadvertisement area depending on whether or not a predetermined patternexists in the virtual viewpoint image. The method of setting theadvertisement area and the method of identifying the advertisement areaare not limited to those described in the embodiment.

Based on the advertisement area information generated from theadvertisement area identifying unit 1401 and the advertisement imagestored in the advertisement image storing unit 408, the advertisementimage converting unit 1402 converts the advertisement image size so asto be held within the range of the advertisement area information, andgenerates the converted advertisement image. Incidentally, in thepresent embodiment, an initial size including the width and the heightof the advertisement image is preset. When the advertisement image sizeis held within the width and the height of the advertisement areainformation generated by the advertisement area identifying unit 1401,the advertisement image converting unit 1402 uses the advertisementimage of the initial size as the converted advertisement image. On theother hand, when the initial size of the advertisement image is not heldwithin the advertisement area indicated by the advertisement areainformation, the advertisement image converting unit 1402 reduces theadvertisement image so as to be held within the advertisement areaindicated by the advertisement area information.

As another example, the advertisement image converting unit 1402 maydetermine the largeness (size) of the advertisement image (virtualadvertisement) based on a positional relationship between thethree-dimensional position of the virtual viewpoint and the position ofthe advertisement area. The process for adjusting the size of theadvertisement image according to the advertisement area is not limitedto that in the embodiment. As another example, the image generatingapparatus 1400 may previously hold a plurality of advertisement imagesof different sizes in the advertisement image storing unit 408, and readout an advertisement image of a size held within the advertisement areabased on the advertisement area information. As just described, themethod of converting the advertisement image is not limited to that inthe embodiment.

The image synthesizing unit 1403 synthesizes the virtual viewpoint imagegenerated by the virtual viewpoint image generating unit 410 and theconverted advertisement image generated by the advertisement imageconverting unit 1402 to generate an advertisement-synthesized virtualviewpoint image. That is, the image synthesizing unit 1403 superimposesthe advertisement image on a designated area based on the advertisementarea information of the virtual viewpoint image, thereby generating theadvertisement-synthesized virtual viewpoint image. Theadvertisement-synthesized virtual viewpoint image generated by the imagesynthesizing unit 1403 is converted into a transmission signal capableof being transmitted by the communication processing unit 401, andtransmitted to the terminal apparatus 130.

FIG. 15 is a flowchart for describing an image process to be performedby the image generating apparatus 1400 according to the fourthembodiment. It should be noted that, among the processes shown in FIG.15, the same processes as those of the image process shown in FIG. 5 aredenoted by the same reference numerals respectively, and the descriptionthereof will be omitted. After the process in S503, in S1501, thevirtual viewpoint image generating unit 410 generates a virtualviewpoint image based on the specific object image and the backgroundimage obtained by the image obtaining unit 405. Next, in S1502, theadvertisement area identifying unit 1401 identifies the advertisementarea in the virtual viewpoint image. Next, in S1503, the advertisementimage converting unit 1402 converts the size of the advertisement imagebased on the advertisement area information generated by theadvertisement area identifying unit 1401 and the advertisement imagestored in the advertisement image storing unit 408, thereby obtainingthe converted advertisement image. The advertisement image convertingunit 1402 reduces the size of the advertisement image such that theadvertisement image is held within the range of the advertisement areainformation.

Next, in S1504, the image synthesizing unit 1403 superimposes(synthesizes) the converted advertisement image obtained in S1503 on thedesignated area of the virtual viewpoint image, thereby generating theadvertisement-synthesized virtual viewpoint image in which theadvertisement is disposed. Next, in S1505, the communication processingunit 401 transmits (outputs) the advertisement-synthesized virtualviewpoint image (display image) obtained in S1504 to the terminalapparatus 130. Thus, the image process ends.

FIG. 16 and FIG. 17 are explanatory diagrams of the advertisement areaidentifying process (S1502). FIG. 16 conceptually shows the virtualviewpoint 600, the virtual capturing range 601, the specific object 602,and an advertisement area 1600. FIG. 17 shows a virtual viewpoint imageobtained from the virtual viewpoint 600 of FIG. 16. For example, in thevirtual viewpoint image obtained from the virtual viewpoint informationof the virtual viewpoint 600, it is assumed that the whole virtualviewpoint image is an area of (0, 0) to (1920, 1080) dots whose originis the upper left. In this case, by the two-dimensional projectiontransformation from the virtual viewpoint space to the virtual viewpointimage, an area of position (0, 100), width 700 and height 300 isidentified as an advertisement area 1700.

FIGS. 18A and 18B are explanatory diagrams of the advertisement imageconverting process (S1503) and the generating process of theadvertisement-synthesized virtual viewpoint image (S1504). As shown inFIG. 18A, it is assumed that an advertisement image 1800 is an imagewith width 720 and height 360. It is assumed that the advertisement areaidentified in S1502 is an area of width 700 and height 300 as shown inFIG. 17. In this case, the advertisement image converting unit 1402reduces the advertisement image to a size which is held within in theadvertisement area while maintaining the aspect ratio of theadvertisement image. More specifically, the converted advertisementimage having width 600 and height 300 is generated. Then, as shown inFIG. 18B, by synthesizing a converted advertisement image 1810 at theposition (0, 100) of the virtual viewpoint image, anadvertisement-synthesized virtual viewpoint image 1820 is generated.

In the present embodiment, the image synthesizing unit 1403 disposes theadvertisement image with the center of the advertisement area as thereference. However, the advertisement image may be disposed within theadvertisement area, and the disposition of the advertisement image isnot limited to that in the embodiment. As another example, the imagesynthesizing unit 1403 may dispose the advertisement image such that theupper left vertex of the advertisement area and the upper left vertex ofthe advertisement image coincide with each other. Besides, theadvertisement image converting unit 1402 does not necessarily need tomaintain the aspect ratio in the conversion of the advertisement imagesize. Namely, the advertisement image converting unit may convert onlyone of the vertical and horizontal directions, or may reduce thevertical and horizontal directions respectively at differentmagnifications.

As described above, the image generation apparatus 1400 according to thefourth embodiment disposes the virtual advertisement image in theadvertisement area included in the virtual viewpoint image. Morespecifically, based on the position and the size of the advertisementarea of the virtual viewpoint image, the image generating apparatus 120converts the advertisement image so as to be within the advertisementarea, synthesizes the virtual viewpoint image and the convertedadvertisement image, and displays the synthesized image. According tothe above constitution, it is possible to add the advertisement imageafter generating the virtual viewpoint image, and it is thus possible toadd the advertisement image with a simpler constitution andconfiguration.

Incidentally, it should be noted that the methods respectively describedin the second and third embodiments may be applied to the fourthembodiment. More specifically, the advertisement image may be convertedbased on a movement direction of the specific object, or a size occupiedby the image of the specific object in the virtual viewpoint image.

Further, the method of controlling the size of the advertisement imageis not limited to the method of directly controlling the size of theadvertisement image in the two-dimensional virtual viewpoint image.Namely, it may possible to control the size of the advertisement imagein the three-dimensional space and generate the virtual viewpoint imagebased on such a control result. For example, in the advertisement areaspredetermined in the three-dimensional space and the advertisement areadetermined based on the virtual viewpoint and the position of thespecific object, the image generating apparatus identifies a partialarea included in the field of view of the virtual viewpoint (a partincluded in the virtual viewpoint image). Such identification of thepartial area is performed based on, for example, the three-dimensionalposition of the virtual viewpoint and the three-dimensional position ofthe advertisement area. Then, the image generating apparatus determinesthe size and the content of the advertisement image such that theadvertisement image is held within the identified partial area, anddisposes the advertisement image in the partial area. Then, the imagegenerating apparatus may generate the virtual viewpoint image based onthe captured image, the virtual viewpoint information, and the position,the size and the content of the advertisement image in thethree-dimensional space. According to such a method, for example, it ispossible to include the whole advertisement image in the virtualviewpoint image, so that an advertisement effect can be enhanced ascompared with a case where only a part of the advertisement image isincluded in the virtual viewpoint image.

Besides, the image generating apparatus may previously hold, in additionto the sizes of the advertisement image, advertisement images differentin advertisement contents in the advertisement image storing unit 408,and may read different advertisement images based on the advertisementarea information. Here, the advertisement content is advertisementinformation including a company name logo, a product logo, a productimage, a logo mark and an image. For example, as the advertisement areabecomes smaller, less advertisement information is read out. As anotherexample, the image generating apparatus may determine the content of thevirtual advertisement based on the positional relationship between thethree-dimensional position of the virtual viewpoint and the position ofthe advertisement area.

In each of the above embodiments, the advertisement image and thevirtual viewpoint image are synthesized by disposing the advertisementarea displaying the advertisement image in the capturing space. However,the image generating apparatus only has to synthesize the virtual objectnot included in the captured image by the image capturing apparatus 110and the virtual viewpoint image, and the virtual object is not limitedto the advertisement image. For example, the virtual object to besynthesized may be additional information such as an image showinginformation related to the progress of a capturing-target game, an imageshowing statistical information related to a game, character and imagedescribing a subject such as a person, an image showing biometricinformation of a subject person, and the like.

As just described, although the examples of the embodiments of thepresent invention have been described in detail, the present inventionis not limited to the specific embodiments. That is, variousmodifications and changes are possible within the scope of the gist ofthe present invention described in the later-described claims.

According to the present invention, it is possible to reduce apossibility that a subject (specific object) of a virtual viewpointimage and a virtual object area overlap each other.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An image processing apparatus comprising: one ormore hardware processors; and one or more memories storing one or moreprograms configured to be executed by the one or more hardwareprocessors, the one or more programs including instructions for: (A)obtaining object information indicating a three-dimensional position anda three-dimensional shape of a specific object in a three-dimensionalspace, wherein the specific object is included in at least one ofcaptured images obtained by a plurality of imaging apparatuses; (B)obtaining viewpoint information indicating a three-dimensional positionof a virtual viewpoint and a direction from the virtual viewpoint in thethree-dimensional space; (C) determining, based on (1) thethree-dimensional shape of the specific object indicated by the obtainedobject information and (2) a positional relationship between (a) thethree-dimensional position of the specific object indicated by theobtained object information and (b) the three-dimensional position ofthe virtual viewpoint indicated by the obtained viewpoint information, athree-dimensional position of a virtual object in the three-dimensionalspace, wherein the virtual object is not included in the plurality ofcaptured images; (D) determining information presented by the content ofthe virtual object based on a size of a region corresponding to avirtual object in a virtual viewpoint image corresponding to theobtained viewpoint information; (E) generating the virtual viewpointimage including the region corresponding to a virtual object whosecontent presents the determined information, based on (i) the obtainedobject information, (ii) the obtained viewpoint information, and (iii)the determined three-dimensional position of the virtual object.
 2. Theimage processing apparatus according to claim 1, wherein thethree-dimensional position of the virtual object is determined so thatan image corresponding to a three-dimensional shape of the specificobject and an image corresponding to the virtual object do not overlapin the virtual viewpoint image.
 3. The image processing apparatusaccording to claim 1, wherein the three-dimensional position in thethree-dimensional space is defined by a three-dimensional coordinatesystem corresponding to a space captured by the plurality of imagingapparatuses.
 4. The image processing apparatus according to claim 3,wherein the three-dimensional position of the virtual object isdetermined such that a distance between the three-dimensional positionof the virtual viewpoint and the three-dimensional position of thevirtual object in a case where a distance between the three-dimensionalposition of the virtual viewpoint and the three-dimensional position ofthe specific object is a first distance is longer than a distancebetween the three-dimensional position of the virtual viewpoint andthree-dimensional position of the virtual object in a case where thedistance between the three-dimensional position of the virtual viewpointand the three-dimensional position of the specific object is a seconddistance that is longer than the first distance.
 5. The image processingapparatus according to claim 1, wherein the one or more programs furtherincludes instructions for detecting a movement direction of the specificobject; and wherein the three-dimensional position of the virtual objectis determined based on the detected movement direction of the specificobject.
 6. The image processing apparatus according to claim 1, whereinthe one or more programs further includes instructions for specifying ashape of another object that is different from the specific object andis included in the at least one captured images, and wherein thethree-dimensional position of the virtual object is determined based onthe specified shape of the another object.
 7. The image processingapparatus according to claim 1, wherein the three-dimensional positionof the virtual object is determined based on a size of the specificobject in the virtual viewpoint image determined according to theobtained object information and the obtained viewpoint information. 8.The image processing apparatus according to claim 1, wherein at leastone of a size of the virtual object and a shape of the virtual object isdetermined based on the obtained object information and the obtainedviewpoint information.
 9. The image processing apparatus according toclaim 1, wherein the size of a region corresponding to the virtualobject in the virtual viewpoint image is determined based on a distancebetween (a) the three-dimensional position of the virtual viewpointindicated by the obtained viewpoint information and (b) the determinedthree-dimensional position of the virtual object.
 10. The imageprocessing apparatus according to claim 9, wherein where (1) a firstcase is a case where a distance between the three-dimensional positionof the virtual viewpoint indicated by the obtained viewpoint informationand the determined three-dimensional position of the virtual object is afirst distance and (2) a second case is a case where a distance betweenthe three-dimensional position of the virtual viewpoint indicated by theobtained viewpoint information and the determined three-dimensionalposition of the virtual object is a second distance that is shorter thanthe first distance, the following condition is satisfied: informationpresented by the content of the virtual object in the first case is lessthan information presented by the content of the virtual object in thesecond case.
 11. The image processing apparatus according to claim 1,wherein the virtual object is an object for displaying an advertisementin the virtual viewpoint image.
 12. The image processing apparatusaccording to claim 11, wherein a content of the virtual object displayedin the virtual viewpoint image includes at least one of a company logo,a product logo, and a product image.
 13. The image processing apparatusaccording to claim 1, wherein the virtual object is an object fordisplaying, in the virtual viewpoint image, additional informationrelated to a capturing target of the plurality of imaging apparatuses.14. The image processing apparatus according to claim 13, wherein acontent of the virtual object displayed in the virtual viewpoint imageincludes at least one of information related to a progress of acapturing-target game, statistical information related to a game,information describing a person, and biometric information of a person.15. The image processing apparatus according to claim 1, wherein thegenerated virtual viewpoint image corresponds to the three-dimensionalposition of the virtual viewpoint and the direction from the virtualviewpoint indicated by the obtained viewpoint information and includesthe virtual object arranged at the determined three-dimensional positionof the virtual object.
 16. The image processing apparatus according toclaim 1, wherein the specific object is an object that moves in a spacecaptured by the plurality of imaging apparatuses.
 17. The imageprocessing apparatus according to claim 16, wherein the specific objectis at least one of a person and a ball in a space captured by theplurality of imaging apparatuses.
 18. An image generating methodexecutable by an image processing apparatus, the method comprising: (A)obtaining object information indicating a three-dimensional position anda three-dimensional shape of a specific object in a three-dimensionalspace, wherein the specific object is included in at least one ofcaptured images obtained by a plurality of imaging apparatuses; (B)obtaining viewpoint information indicating a three-dimensional positionof a virtual viewpoint and a direction from a viewpoint in thethree-dimensional space; (C) determining, based on (1) thethree-dimensional shape of the specific object indicated by the obtainedobject information and (2) a positional relationship between (a) thethree-dimensional position of the specific object indicated by theobtained object information and (b) the three-dimensional position ofthe virtual viewpoint indicated by the obtained viewpoint information, athree-dimensional position of a virtual object in the three-dimensionalspace, wherein the virtual object is not included in the plurality ofcaptured images; (D) determining information presented by the content ofthe virtual object based on a size of a region corresponding to avirtual object in a virtual viewpoint image corresponding to theobtained viewpoint information; and (E) generating the virtual viewpointimage including the region corresponding to a virtual object whosecontent presents the determined information, based on (i) the obtainedobject information, (ii) the obtained viewpoint information, and (iii)the determined three-dimensional position of the virtual object.
 19. Theimage generating method according to claim 18, wherein thethree-dimensional position of the virtual object is determined so thatan image corresponding to a three-dimensional shape of the specificobject and an image corresponding to the virtual object do not overlapin the virtual viewpoint image.
 20. A non-volatile computer-readablestorage medium which stores a program for causing a computer to executea method comprising: (A) obtaining object information indicating athree-dimensional position and a three-dimensional shape of a specificobject in a three-dimensional space, wherein the specific object isincluded in at least one of captured images obtained by a plurality ofimaging apparatuses; (B) obtaining viewpoint information indicating athree-dimensional position of a virtual viewpoint and a direction from avirtual viewpoint in the three-dimensional space; (C) determining, basedon (1) the three-dimensional shape of the specific object indicated bythe obtained object information and (2) a positional relationshipbetween (a) the three-dimensional position of the specific objectindicated by the obtained object information and (b) thethree-dimensional position of the virtual viewpoint indicated by theobtained viewpoint information, a three-dimensional position of avirtual object in the three-dimensional space, wherein the virtualobject is not included in the plurality of captured images; (D)determining information presented by the content of the virtual objectbased on a size of a region corresponding to a virtual object in avirtual viewpoint image corresponding to the obtained viewpointinformation; and (E) generating the virtual viewpoint image includingthe region corresponding to a virtual object whose content presents thedetermined information, based on (i) the obtained object information,(ii) the obtained viewpoint information, and (iii) the determinedthree-dimensional position of the virtual object.